Method and system for automatic technical assistance

ABSTRACT

A method and system for automatic technical assistance is provided. The technical assistant is first trained using the user profile of the users who will be accessing the technical assistant, and the workflow details. The technical assistant continuously monitors user screen and checks if the user is on the correct screen for the required workflow. Based on user confirmation an RPA bot resolves the error for the user. In an embodiment, the bot completes the workflow for the user. The bot may also provide guidance content to the user to help the user realize the error.

This application claims the benefit of Indian Patent Application SerialNo. 202041042524 filed Sep. 30, 2020, which is hereby incorporated byreference in its entirety.

FIEL

This technology generally relates to technical assistance for users.More specifically it relates to an automated technical assistance.

BACKGROUND

In traditional support operations, A user when faced with an issue witha workflow needs to contact help desk either via a call or log a ticketthrough a portal. Some of the automated assistant automated technicalassistant agent configured to receive request to initiate technicalsupport, communicate response, receive selection of the one of theresponses, and provide course of action to the user. However, a userassistant to understand user profile, user intent, and issue will beable to resolve and thereby provide faster resolution and greater userexperience.

SUMMARY

A method for providing automatic technical assistance, comprisingproviding a set of parameters relating to each user from a set of users,a set of workflows, and a data relating to the set of workflows to thetechnical assistant. It further comprises receiving a request from auser from the set of users for technical assistance regarding a workflowfrom the set of workflows, assisting the user by resolving the error bythe technical assistant, wherein the resolving comprises periodicallycapturing an image of a screen of a user device, processing each of thecaptured image to identify if the user is on a right page for theworkflow, using the provided set of workflows and the data relating tothe set of workflows, processing a set of inputs for the workflow,provided by the user, to identify an error in the set of inputs, usingthe provided set of parameters relating to the user, and assisting theuser based on the above processing and navigating the user to completethe workflow; or based on user confirmation perform one of completingthe workflow by the technical assistant, or providing a guiding contentto the user to resolve the error.

A system for providing automatic technical assistance, comprising one ormore processors, a memory coupled to one or more processors which areconfigured to execute programmed instructions stored in the memory whichcomprises providing to the technical assistant, a set of parametersrelating to each user from a set of users, a set of workflows, and adata relating to the set of workflows; receiving a request from a userfrom the set of users, for technical assistance regarding a workflowfrom the set of workflows; assisting the user by resolving the error bythe technical assistant, wherein the resolving comprises periodicallycapturing an image of a screen of a user device, processing each of thecaptured image to identify if the user is on a right page for theworkflow, using the provided set of workflows and the data relating tothe set of workflows, processing a set of inputs for the workflow,provided by the user, to identify an error in the set of inputs, usingthe provided set of parameters relating to the user; and assisting theuser based on the above processing and navigating the user to completethe workflow; or based on user confirmation perform one of completingthe workflow by the technical assistant, or providing a guiding contentto the user to resolve the error.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary network environment

FIG. 2 is a flowchart of an exemplary method for implementing thetechnical assistant;

FIG. 3 is a detailed flowchart of an exemplary method for implementingthe technical assistant;

FIG. 4 is an illustration of the architecture for implementing thetechnical assistant; and

FIG. 5 is an exemplary technical assistant as per the disclosure.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a computing device 100 to which the presentdisclosure may be applied according to an embodiment of the presentdisclosure. The system includes at least one processor 102, designed toprocess instructions, for example computer readable instructions (i.e.,code) stored on a storage device 104. By processing instructions,processing device 102 may perform the steps and functions disclosedherein. Storage device 104 may be any type of storage device, forexample, but not limited to an optical storage device, a magneticstorage device, a solid state storage device and a non-transitorystorage device. The storage device 104 may contain software 104a whichis a set of instructions (i.e. code). Alternatively, instructions may bestored in one or more remote storage devices, for example storagedevices accessed over a network or the internet 106. The computingdevice also includes an operating system and microinstruction code. Thevarious processes and functions described herein may either be part ofthe microinstruction code or part of the program (or combinationthereof) which is executed via the operating system. Computing device100 additionally may have memory 108, an input controller 110, and anoutput controller 112 and communication controller 114. A bus (notshown) may operatively couple components of computing device 100,including processor 102, memory 108, storage device 104, inputcontroller 110 output controller 112, and any other devices (e.g.,network controllers, sound controllers, etc.). Output controller 110 maybe operatively coupled (e.g., via a wired or wireless connection) to adisplay device (e.g., a monitor, television, mobile device screen,touch-display, etc.) in such a fashion that output controller 110 cantransform the display on display device (e.g., in response to modulesexecuted). Input controller 108 may be operatively coupled (e.g., via awired or wireless connection) to input device (e.g., mouse, keyboard,touch-pad, scroll-ball, touch-display, etc.) in such a fashion thatinput can be received from a user. The communication controller 114 iscoupled to a bus (not shown) and provides a two-way coupling through anetwork link to the internet 106 that is connected to a local network116 and operated by an internet service provider (hereinafter referredto as ‘ISP’) 118 which provides data communication services to theinternet. Network link typically provides data communication through oneor more networks to other data devices. For example, network link mayprovide a connection through local network 116 to a host computer, todata equipment operated by an ISP 118. A server 120 may transmit arequested code for an application through internet 106, ISP 118, localnetwork 116 and communication controller 114. Of course, FIG. 1illustrates computing device 100 with all components as separate devicesfor ease of identification only. Each of the components may be separatedevices (e.g., a personal computer connected by wires to a monitor andmouse), may be integrated in a single device (e.g., a mobile device witha touch-display, such as a smartphone or a tablet), or any combinationof devices (e.g., a computing device operatively coupled to atouch-screen display device, a plurality of computing devices attachedto a single display device and input device, etc.). Computing device 100may be one or more servers, for example a farm of networked servers, aclustered server environment, or a cloud network of computing devices.

Although an exemplary computing device 100 and its components aredescribed and illustrated herein, other types and numbers of systems,devices in other topologies can be used. It is to be understood that thesystems of the examples described herein are for exemplary purposes, asmany variations of the specific hardware and software used to implementthe examples are possible, as will be appreciated by those skilled inthe relevant art(s).

Furthermore, each of the systems of the examples may be convenientlyimplemented using one or more general purpose computer systems,microprocessors, digital signal processors, and micro-controllers,programmed according to the teachings of the examples, as described andillustrated herein, and as will be appreciated by those of ordinaryskill in the art.

The examples may also be embodied as a non-transitory computer readablemedium having instructions stored thereon for one or more aspects of thetechnology as described and illustrated by way of the examples herein,which when executed by a processor (or configurable hardware), cause theprocessor to carry out the steps necessary to implement the methods ofthe examples, as described and illustrated herein.

An exemplary method for providing automatic technical assistant willbroadly be described along with FIG. 2. In step 201 the technicalassistant is first trained using the user profile of the users who willbe accessing the technical assistant. The user profile may contain theaccess rights associated with the user. In another embodiment the userprofile may also contain the workflow policies associated with the user.The technical assistant is also trained with the workflows and the itsprocesses which will be used by the users. The screens of the workflowsin the user system, are used to train the technical assistant.

In step 202 a user raises a request for technical assistance for anerror he got while using a workflow. If user confirms for liveassistance, the technical assistant asks the user to share his screen.The technical assistant then checks is the user is on the correct screenfor the required workflow.

In step 203, the technical assistant periodically captures the userscreens while the user tries to access the workflow. The technicalassistant keeps analyzing the screens captured, and informs the userwhen he makes an error leading to the error. Message.

In step 204, alternatively, if user confirms for error resolving, an RPAbot resolves the error for the user. In an embodiment, the bot completesthe workflow for the user. The bot may also provide guidance content tothe user to help the user realize the error. In an embodiment theguidance content may include video, documents, image files or any otherappropriate format.

An exemplary method for providing automatic technical assistant will bedescribed in detail along with FIG. 2.

For any given application in a system, there can be many workflows orfunctional flows and that becomes the standard operation procedures.Every workflow can have one or more steps and every steps can have oneor more corresponding user screens in a user device. Some of theworkflows can be happy workflows or unhappy workflows. For e.g. In abanking application, when you want to transfer funds from account A toaccount B, happy scenarios is when you are able to successfully transferfunds from A to B. If there is a failure in the workflow such as unableto find payee, unable to transfer funds, or insufficient funds are partof unhappy workflows. Each of the happy and unhappy workflows can bethought of a workflow for the technical assistant and having their ownflows of screens.

The bot (300) is trained initially with the workflow details. Thedetails of the workflow may include the workflow steps and correspondingscreens of the user interface, from the user's device. The technicalassistant may have access to user profile information to identify theuser access levels or rights in a system, priori transaction informationof the user, intent of the user to determine what the user is trying todo and if the action is allowed. In another embodiment, all theworkflows and their respective steps and screens of an application arestored along with additional details such as resolutions steps, errorinformation, NER information, RPA availability, etc. These details maybe stored in a repository accessible the technical assistant. Thescreens are stored in vectorized form. Any known computer visiontechnology can be used to convert the screens from the user interface.In one embodiment, the technical assistant and the training detailsmaybe saved in a repository on the server. This may also be saved on thecloud.

In one embodiment, named entity recognition or any other technology forentity recognition, is used on the captured screen to save all thepossible labels and the respective patterns for the functionalities inan application.

The resolution of the errors maybe pre-saved with the technicalassistant in a database. It will provide the solutions to the user afterreferring the error in the database.

In an embodiment, when a user faces an error while using a workflow inan application, he initiates the technical assistant (300). Thetechnical assistant provides options to the user and asks him to choose.Technical assistant may ask if the user wants an error resolution, or ifan RPA process may complete the workflow on behalf of the user (310).

Each option and the subsequent process will now be described further.The first description will be for the user selecting the option of errorresolution (312). In an embodiment, this option uses live sharing of theuser screen. The user shares his screen with the technical assistant(312.1). In one embodiment the technical assistant accesses user screenby capturing images of the user screen (312.2). The technical assistantcan use any existing technology to capture the user screens.

The user performs the workflow and the technical assistant analyses theuser screen in real time, continuously. The captured screen maybecaptured, processed and transferred to the required technology andimplementation platform as required by the user. The captured screenfrom the user screen maybe converted to vector form. These images arecompared with the vector form of the images initially saved in therepository, while training the technical assistant. In an embodiment,when a screen image is captured, the technical assistant will convertthat into a vector and finds the Euclidean distance with the initiallysaved vectors. The technical assistant also extracts the labels and textfrom the captured image and match the pattern in extracted text with thestored labels. All this is done in real time and continuously as theuser progresses to further steps in the workflow.

In one embodiment, the technical assistant may process the image beforeanalyzing and matching with the image vectors stored initially. Thetechnical assistant may add pixels to the boundaries of the image,adjust the light intensity of the image, remove the anomalies and noisefrom the image. The technical assistant may use any other imageprocessing steps to improve and change the image quality as needed.

In an embodiment, the technical assistant then uses text extraction toanalyses the labels and text in the captured image. This can be donethrough Optical Character Recognition or Named Entity Recognition or anyother appropriate technology that can be used for text extraction. Theextracted text is then compared with the text and labels storedinitially while training the technical assistant.

In an embodiment, the text matching maybe done using regularexpressions. Any other technology that provides text matching can alsoused by the technical assistant for matching the text extracted from thecaptured images.

The technical assistant may perform the image processing, textextraction and matching in real time, as the user keeps progressing withthe workflow. The technical assistant may capture the screen image asper a time interval set by the user. After every preconfigured timeinterval, e.g. 5 seconds, the technical assistant may capture an image,process the image, extract the text and perform image and text matching.

In an embodiment, the technical assistant maybe trained with userdetails. As previously mentioned, the technical assistant may haveaccess to user profile information to identify the user access levels orrights in a system, priori transaction information of the user, intentof the user to determine what the user is trying to do and if the actionis allowed. The text extracted from the captured user screen is analyzedand matched with the user access and levels stored with the technicalassistant. Accordingly, the technical assistant can perform all matchingand conclude if the user has navigated to wrong screen is there is ascreen mismatch, or if there is user access error.

As the user navigates through the workflow after initiating thetechnical assistant and confirming the screen share, the technicalassistant captures the user screen periodically, processes and analyseit. Based on the screen matching, if the technical assistant concludesthat the user is at the right workflow for his intent, the user tries toextract the text as described above and use the text, labels and userprofile details saved with the technical assistant. Hereby the technicalassistant can identify if the user has given wrong entry on the page orscreen. The user may have tried a workflow which is not accessible tohim, or the user may have provided invalid inputs.

In an embodiment, where the technical assistant identifies that the textinput provided by user is invalid, it may provide a notification to theuser about the error and help resolve the error. The technical assistantmay provide a resolution message also to the user. Accordingly the usercan decide the next steps or if he wishes to continue with the workflow.In one embodiment, the error can be resolved by triggering an RPA tocomplete the process. The RPA may complete the workflow on behalf of theuser. It may prompt the user to choose a correct option to provide forthe workflow. In another embodiment, it can be any other function orprocess configured to take appropriate inputs from the user and completethe workflow. If the technical assistant is unable to identify the errorafter screen match and text match, then a ticket is raised with abackend technical assistant team.

In one embodiment, if the technical assistant identifies after a screenmatch that the user is at the wrong screen, or a wrong workflow of aStandard Operating Procedure, he may be directed to the right workflow.Alternatively, the user maybe provided a guidance artifact suggestingthe right workflow.

The description will now elaborate the technical assistant process ifthe user initially selects RPA process instead of error resolution. Inan embodiment the RPA process confirms from the user if RPA processshould finish the workflow. Once confirmed the RPA process may take thevalues to be entered from the user and finish the process appropriately.In an embodiment if the RPA process faces an error while completing theworkflow for the user, the technical assistant may extract the text fromthe screen and match with the text details stored with the technicalassistant. The technical assistant may be notified about the error inthe text entered or it may be directed to raise a ticket with thebackend team of the technical assistant.

If the user selects that he does not want the RPA process to finish theworkflow, then the user may be guided to finish the workflow usingappropriate tutorials. The tutorials are configurable as per the userrequirements, the workflow, and the screens. In one embodiment, as theuser completes the workflow using the tutorial the technical assistantcontinuously monitors the screen.

An exemplary architecture for implementing the technical assistant willbe explained along with FIG. 4. In an embodiment the technical assistantis integrated with appropriate components for implementing the technicalassistant and various processes associated with the technical assistant.The components can be installed locally in the user machine or in theserver machine along with the technical assistant, or can be remotelyaccessed in any other connected machine.

In an embodiment, the technical assistant maybe configured to beintegrated with an ITSM tool. The ITSM tool enables the technicalassistant to raise and service requests as needed by the technicalassistant.

In one embodiment, the technical assistant maybe configured to beintegrated with RPA process. The RPA process may be triggered when auser selects the option of RPA assistance. In one embodiment, the RPAprocess may be triggered when the technical assistant is unable toidentify the error by screen sharing. The technical assistant may beconfigured to trigger the RPA at any other appropriate time of userassistance.

In one embodiment the technical assistant may be configured to integratewith the notifying components to notify the user as appropriate. Thenotifying component can be a mail exchange server, or a messagingserver, or any other component as appropriate. The user may be notifiedin case a ticket is raised for the error faced by the user. The user maybe notified when a ticket is resolved. The technical assistant may beconfigured to notify the user at any other appropriate time.

In one embodiment the technical assistant maybe integrated withconfigurable databases and repository. These data storage can be cloudbased, custom database, file repositories or any other database. Thesecan be used for storing any structure of data. The technical may beconfigured to store screens in vector form, text extracted from screens,user profiles and other related data. These data can be retrieved andprocessed during the process of user assistance. These data storage canalso be configured to store the error resolution details, correspondingto various possible errors in a workflow.

The technical assistant can be integrated with any other appropriatecomponents for the configurable implementation. The components can beintegrated using any known communication means. Other appropriatecomponents maybe audio components, user interface components, analyticscomponents and other such components.

In one embodiment, the components of the architecture for implementingthe technical assistant can be installed and integrated through separatesystems in a network or in a cloud ecosystem.

An exemplary technical assistant as per the disclosure will be explainedalong with FIG. 5. The explanation given below is only by way of examplein one user scenario, for the purpose of illustration. This should notbe considered limiting by any means.

In the present example, a user tries to apply for leaves in hisorganization. The user faces error while applying leaves. The errormaybe that ‘Leave cannot be applied’. The user initiates the technicalassistant to help resolve the error.

If the user opts for error resolving, the technical assistant asks userif it can access the user screen. The technical assistant then capturesthe user screen continuously as the user progresses through the leaveapplication workflow. For the purpose of this example, we can considerthat the technical assistant uses Flask platform to capture the screen.The technical assistant keeps converting the captured into vector formsand compare with the vector details available in its repository. OneShotlearning converts the screen into vector form. In this example, if theuser is on the correct screen, the technical assistant will get acorrect match and notify that the user is on ‘LeaveApplication’ screen.

Next the technical assistant will extract the data from the capturedscreens and use appropriate technology to convert them into text. Inthis example, the technical assistant uses Tesseract platform to extractthe text from the screen and match with its database. Once the technicalassistant performs the text matching for the captured screens, it comesacross the screen where the user has put dates for the leave. It mayfind an error in dates and give an Error Message—‘Leave period cannotbegin from a non-working day’. The technical assistant may also give aResolving Message—‘Please apply leave on any working day’.

Accordingly the technical assistant also provides a list of relatedparameters, that may be number of leaves available, the name of leaveapprover or the type of leave to be selected.

The user may select RPA bot assistant to complete the workflow. The botwill take leave dates from the user and apply the leaves for the user.If the user confirms, the technical assistant may provide guidingdocument or a video to the user so that user can complete the workflowon his own. Alternatively, technical assistant can raise a ticket if theerror is not resolved.

Having thus described the basic concept of the invention, it will berather apparent to those skilled in the art that the foregoing detaileddisclosure is intended to be presented by way of example only and is notlimiting. Various alterations, improvements, and modifications willoccur and are intended to those skilled in the art, though not expresslystated herein. These alterations, improvements, and modifications areintended to be suggested hereby, and are within the spirit and scope ofthe invention. Additionally, the recited order of processing elements orsequences, or the use of numbers, letters, or other designationstherefore, is not intended to limit the claimed processes to any orderexcept as may be specified in the claims. Accordingly, the invention islimited only by the following claims and equivalents thereto.

What is claimed is:
 1. A method for providing automatic technicalassistance, the method comprising: providing to a technical assistant, aset of parameters relating to each user from a set of users; a set ofworkflows; and a data relating to the set of workflows; receiving arequest from a user from the set of users, for technical assistanceregarding a workflow from the set of workflows; assisting the user by:resolving the error by the technical assistant, wherein the resolvingcomprises: periodically capturing an image of a screen of a user device;processing each of the captured image to identify if the user is on aright page for the workflow, using the provided set of workflows and thedata relating to the set of workflows; processing a set of inputs forthe workflow provided by the user, to identify an error in the set ofinputs, using the provided set of parameters relating to the user; andassisting the user based on the above processing and navigating the userto complete the workflow; or based on user confirmation, perform one of:completing the workflow by the technical assistant; or providing aguiding content to the user to resolve the error.
 2. The method asclaimed in claim 1, further comprising creating a ticket for the userfor resolving the error.
 3. The method as claimed in claim 1, whereinthe set of parameters relating to each user comprises an identifier forthe user, the access available to the user, and a set of one or morepolicies related to the user.
 4. The method as claimed in claim 1,wherein the data relating to the set of workflows comprises a processfor each workflow from the set of workflows, and one or more relatedscreens for each of the process.
 5. The method as claimed in claim 1,wherein the processing the captured image comprises adjusting thebrightness of the image using pixels, removing anomalies from thecaptured image, and removing noise from the captured image.
 6. Themethod as claimed in claim 4, wherein the set of input parameters areextracted from the processed image and compared with the provided set ofparameters relating to each user, to identify the error.
 7. The methodas claimed in claim 1, wherein the guiding content comprises documentand video files.
 8. A system for providing automatic technicalassistance, comprising: one or more processors; a memory coupled to oneor more processors which are configured to execute programmedinstructions stored in the memory comprising, providing to a technicalassistant, a set of parameters relating to each user from a set ofusers; a set of workflows; and a data relating to the set of workflows;receiving a request from a user from the set of users, for technicalassistance regarding a workflow from the set of workflows; assisting theuser by: resolving the error by the technical assistant, wherein theresolving comprises: periodically capturing an image of a screen of auser device; processing each of the captured image to identify if theuser is on a right page for the workflow, using the provided set ofworkflows and the data relating to the set of workflows; processing aset of inputs for the workflow, provided by the user, to identify anerror in the set of inputs, using the provided set of parametersrelating to the user; and assisting the user based on the aboveprocessing and navigating the user to complete the workflow; or based onuser confirmation, perform one of: completing the workflow by thetechnical assistant; or providing a guiding content to the user toresolve the error.
 9. The system as claimed in claim 8, furthercomprising creating a ticket for the user for resolving the error. 10.The system as claimed in claim 8, wherein the set of parameters relatingto each user comprises an identifier for the user, the access availableto the user, and a set of one or more policies related to the user. 11.The system as claimed in claim 8, wherein the data relating to the setof workflows comprises a process for each workflow from the set ofworkflows, and one or more related screens for each of the process. 12.The system as claimed in claim 8, wherein the processing the capturedimage comprises adjusting the brightness of the image using pixels,removing anomalies from the captured image, and removing noise from thecaptured image.
 13. The system as claimed in claim 11, wherein the setof input parameters are extracted from the processed image and comparedwith the provided set of parameters relating to each user, to identifythe error.
 14. The system as claimed in claim 1, wherein the guidingcontent comprises document, and video files.
 15. A non-transitorycomputer readable medium having stored thereon instructions forproviding automatic technical assistance, comprising machine executablecode which when executed by at least one processor, causes the at leastone processor to perform steps comprising: providing to a technicalassistant, a set of parameters relating to each user from a set ofusers; a set of workflows; and a data relating to the set of workflows;receiving a request from a user from the set of users, for technicalassistance regarding a workflow from the set of workflows; assisting theuser by: resolving the error by the technical assistant, wherein theresolving comprises: periodically capturing an image of a screen of auser device; processing each of the captured image to identify if theuser is on a right page for the workflow, using the provided set ofworkflows and the data relating to the set of workflows; processing aset of inputs for the workflow, provided by the user, to identify anerror in the set of inputs, using the provided set of parametersrelating to the user; and assisting the user based on the aboveprocessing and navigating the user to complete the workflow; or based onuser confirmation, perform one of— completing the workflow by thetechnical assistant; or providing a guiding content to the user toresolve the error.